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introduction 

President  Bush,  on  September  29,  1989,  in  an  attempt  to  spur  American  competitiveness 
in  the  US  and  foreign  markets,  iaunched  a  campaign  to  implant  management  practices  that 
replicate  the  Total  Quality  Management  philosophy,  fashioned  by  Dr.  W.  Edwards  Deming.  The 
President  stated:  'Reasserting  our  leadership  position  will  require  a  firm  commitment  to  total 
quality  management  and  the  principle  of  continuous  quality  improvement.. .Quality  improvement 
principles  apply  to  small  companies  as  well  as  large  corporations,  to  service  industries  as  well 
as  manufacturing,  and  the  public  sector  as  well  as  private  enterprise.*^  Also,  in  1989,  Air 
Force  Systems  Command  began  an  endeavor  to  instill  this  philosophy,  as  a  business  practice,  in 
the  acquisition  process.  Subsequently,  the  Air  Force  Development  Test  Center  and  the  3246th 
Test  Wing,  located  at  Eglin  AFB,  Florida,  adopted  the  Systems  Command  thrust  into  the  daily 
process  of  Research,  Development,  Test,  and  Evaluation  (RDT&E)  of  nonnuclear  munitions  and 
advanced  development  weapons  systems  testing,  conducted  to  support  the  Life  Cycle  Cost 
acquisition  activities.  Such  testing  encompasses  the  pre-/post-developmental  evaluation  of 
such  weapon  systems  as:  bombs;  missiles;  avionics  systems  and  systems  associated  with 
advanced  development  of  global  navigation;  plasma  energy  devices  and  infrared  sensors; 
perimeter  security  systems  related  to  intrusion  detection  and  personnel  identification  checking 
systems;  and  a  host  of  other  systems  that  are  difficult  to  test.  These  require  multiple  testing 
methodologies  that  are  costly  to  the  government,  but  are  paramount  to  national/global 
security. 

As  part  of  the  endeavor  to  propagate  a  new  management  style  that  would  foster  improved 
work  productivity,  efficiency  and  effectiveness,  the  3246th  Test  Wing  conducted  a  Customer 
Satisfaction  Assessment  to  determine  what  was  important  to  the  customer  (without  secopd 
guessing  them)  and  to  determine  their  likes  and  dislikes  of  the  Wing's  capability  to  meet 
requirements  and  expectations.  The  1991  customer  survey  results  became  the  guiding 
principles  by  which  the  Test  Wing  conducts  business  and  were  integrated  into  the  organizations 
strategic  business  plan,  which  is  maintained  by  its  Total  Quality  Management  (CTQ)  Office. 


Combined  '91  &  '92  Customer  Survey  Ranking 


- ►  5  1991:  37  Responses 

Least  Important  1 992: 1 03  Responses 


*91  Ranking  *92  Ranking 

Figure  1 
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Figure  1  depicts  the  composite  results  obtained  for  years  1991  and  1992.  The  customers 
include  other  U.S.  and  foreign  government  agencies  and  the  Department  of  Defense  (DoD)  who 
are  also  attempting  to  instill  the  "Deming  Philosophy*  of  management  practices  within  their 
own  service  organizations.  It  is  readily  apparent,  from  Figure  1,  that  customers  have  ranked, 
over  the  past  two  years,  the  Test  Execution  Process  as  the  most  important  aspect  of  work 
conducted  on  the  Eglin  complex.  And,  given  that  the  ratings  over  the  last  two  years  have  not 
changed  dramatically,  nor  are  they  pegged  at  the  Very  Satisfied  level,  the  Wing  still  has  some 
work  to  do. 

One  of  the  major  aspects  concerning  the  Test  Execution  Process  of  RDT&E  scheduling  is 
resource  scheduling.  Within  the  Test  Wing,  this  involves  the  utilization  of  scientific  and 
technical  information  of  test  resources,  test  item  capabilities,  etc.,  such  that  properly 
scheduled  missions  will  result  from  accurate  resource  requirements  interpretation  and  testing 
limitations.  This  is  an  area  seen  by  the  customer  as  needing  improvement.  In  this  light. 
Scientific  and  Technical  Information  Command,  Control,  and  Communication  Management 
(STIC^MO)^  can  be  viewed  as  a  field  of  study,  occupying  the  space  where  management  and 
technology  overlap.  STIC^M  technologies,  used  in  conjunction  with  Artificiai 
Intelligence/Expert  Systems  are  today  not  only  impacting  the  processing  of  data,  but  also 
influencing:  (1)  the  way  people  work  at  their  jobs,  (2)  the  quality  of  products,  (3)  how 
decisions  are  made,  and  (4)  the  manner  in  which  many  organizations  compete. 

STIC^M  can  be  conceptualized  as  an  antecedent  to  how  managers  and  executives  should 
view  their  tasks  of  manipulating  and  using  data/information  for  quantitative  and  qualitative 
analyses  in  decision  making.  Information  when  visualized  as  a  key  resource  that  should  be 
managed  strategically  (as  other  resources  through  STIC^M  manipulation  available  for  everyday 
operation),  can  be  facilitated  by  expert  systems  to  assist  in  the  data  analysis  required  for 
decision  making  and  strategic  planning.  This  paper  is  written  in  hopes  of  enlightening  those 
managers  not  familiar  with  the  strategic  importance  of  information  as  a  resource  and  the 
methodologies,  like  expert  systems,  avaiiable  to  them  for  use  to  enhance  their  decision  making, 
through  a  STIO^M  example  case  study. 


Case  Study  ^ 

Test  Wing  Core  Purpose 

The  Wing's  core  purpose  is  to  "satisfy  customer  needs  with  world  class  test  and 
evaluation  of  armament,  electronic  combat,  guidance,  and  other  specialized  DoD  systems."  The 
tenet  here  is  the  organization  will  efficiently  and  cost  effectively  evaluate  each  system  and 
subsystem  to  determine  weapon  system  capability,  reliability,  and  maintainability.  Also, 
RDT&Es  major  thrust  is  to  detect  (and  report  to  the  customer)  early  deficiencies;  requiring 
system  design  changes,  and  to  determine  preparedness  for  production. 

Problem  Definition 

A  major  portion  of  the  support  required  to  conduct  the  RDT&E  mission  is  the 
scheduling  of  resources  essential  to  the  management  of  the  test  and  evaluation.  One  of  the 
problems  associated  with  the  scheduling  of  resources  (to  conduct  individual  test  mission 
objectives)  is  how  to  optimally  schedule  the  maximum  number  of  missions  and  only  those  test 
resources  essential  for  an  effective  evaluation.  This  problem  is  further  compounded  by  the  fact 
that  there  are  50-60  missions  per  day  bidding  for  those  same  resources  and  all  tests  are  on 
"critical  path  time-lines,"  due  to  future  production  and  budgeting  constraints,  that  require 
expeditious  scheduling  and  objective  accomplishment,  . 

The  test  scheduling  cycle  is  accomplished  via  a  dynamic  computer  programming 
model,  developed  in-house,  to  equitably  schedule  all  missions  that  can  optimally  run  at  the  same 
time  and  utilize  only  those  resources  necessary  to  accomplish  their  specific  task;  thus,  leaving 
the  under  utilized  resources  for  lower  priority  test  missions.  This  program  is  referred  to  as 
the  Resource  Scheduling  and  Operations  Management  System  (RESOMS).  Unfortunately, 
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situations  dictate  that  lower  priority  missions,  or  missions  with  special  emphasis,  must  be 
infused  into  the  daily  schedule,  such  that,  the  reliable  completion  of  some  specific  "special" 
tasking  can  be  accomplished.  This  situation  occurs  approximately  three  to  five  times  per  day, 
requires  approximately  2-hours  of  research  per  mission  (which  takes  Test  Wing  Deputy 
Commanders  and  Staff  out  of  pocket  to  do  the  research),  and,  due  to  time  constraints,  must  be 
completed  before  1500-hours  for  inclusion  in  with  other  RESOMS  test  project  requirements. 

As  a  result  of  this  perturbation  of  the  previously  established  schedule  (normally  a 
5-day  time-line),  other  missions  may  be  jeopardized  by  the  loss  of  essential  resources  or  could 
be  totally  eliminated,  to  allow  for  the  incorporation  of  this  so  called  "blackboard"  (BB) 
mission.  (NOTE:  origin  of  the  term  "blackboard"  is  unknown,  but  is  assumed  to  mean  the  bleak 
attitude  associated  with  having  to  change  a  daily  schedule.) 

The  problem  for  management  is  that  accurate  decisions  are  essential  for  determining 
if  a  BB  mission  should  be  scheduled  into  the  daily  activity  (either  allowing  the  BB  to  ride  on  its 
own  precedence  level  or  by  receiving  management  emphasis  to  be  scheduled  "at  any  cost"). 

Management  decisions  are  based  on  the  timely  analysis  of  which  resources  n'e 
needed,  which  resources  are  being  used  by  previously  scheduled  missions,  the  engineer's 
interpretation  of  the  criticality  and  future  testing  direction,  and  what  probable  productivity  can 
be  expected  by  the  mission  being  considered  versus  missions  established  in  the  schedule.  (The 
probable  productivity  being  based  on  such  "fuzzies"  as:  weather,  anticipated  test  item 
operation,  resource  reliability  and  availability,  etc.) 

GURU®  Selection 

The  selection  of  the  expert  system  software  shell  GURU®®  was  made  since  it  offered 
an  environment  for  building  different  expert  system  applications,  the  ability  to  retrieve  and 
present  data  through  natural  language  processing,  and  the  key  essentials  for  contemporary 
business  computing  (i.e.,  graphics,  word  processing,  data  processing,  statistical  analysis, 
etc.).  GURU  allows  the  user  to  create  rule-sets  related  to  certain  topics  and  during 
consultation  GURU  surveys  one  or  more  of  these  sets  to  make  the  proper  recommendation. 
Should  additional  information  be  necessary  to  provide  an  answer,  GURU  asks  for  it  and  can 
explain  why  it  was  needed.  ^ 

GURU'S  reasoning  power  can  use  forward-chaining  to  discover  the  logical  outcome  of 
a  situation,  or  backward-chaining  to  investigate  a  specific  possible  outcomes.  Numerous  search 
strategies  can  be  used  to  determine  which  rules  should  be  considered  for  firing,  how  hard  GURU 
should  work  to  find  values  for  unknown  variables,  and  has  the  ability  to  deal  with  uncertainty 
and  fuzzy  variable  situations.  Rule  sets  can  incorporate  these  "questionable  accuracies,"  which 
enhances  GURU'S  ability  to  make  realistic  recommendations  from  available  information  or  data 
from  internal  or  other  external  databases.  Plus,  GURU  has  the  ability  to  help  check  the  validity 
of  its  conclusions  by  explaining  its  reasoning  process  step-by-step.^ 

Resource  Scheduling  Information  Decision  Support  System  Development 

The  Resource  Scheduling  Information  Decision  Support  System  GURU  prototype, 
developed  by  this  author,  referred  to  as  RESIDS,  will  support  the  timely  analysis  of  the  BB 
situation,  assist  management  in  making  accurate  decisions  given  specific  information  on 
resource  status  and  mission  requirements,  decrease  the  likelihood  of  error,  facilitate  the  BB 
process  (by  taking  the  tedium  out  of  the  task  associated  with  the  analysis  of  alternatives),  and 
allow  the  test  engineer  greater  involvement  in  the  BB  process  and  greater  Input  to  future 
testing. 

Prototype  Development  and  RESIDS  Analysis 

Figure  2  depicts  the  critical  BB  premise  elements,  variables  and  associated 
databases,  and  represents  the  dependency  diagram  initially  visualized  as  the  developmental  tool 
for  rule-sets  and  for  analysis  of  the  overall  goal  attainment.  The  critical  elements,  for  the  BB 
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goal,  requiring  analysis  are:  Range(s)/Resource(s),  Test  Item  Status,  Scheduler  Capability  to 
manipulate  the  RESOMS  program.  Test  Project  Information,  RESOMS  Status/Forecast  of  prior 
and  future  testing  activities  for  the  BB  mission.  Weather,  Management  Action  (for  special 
emphasis),  and  Aircraft  Requirements. 


ype  Aircraft  (RESOMS) 
laintenance  Status  (RESOMS) 
istorical  Reliability  (VAX  11780) 
anker  Support  (RESOMS) 

‘aircrew  Availability  (RESOMS) 
Configuration  Requirements  (RESOMS) 


Figure  2 


CRITICAL  BLACKBOARD  PREMISE  ELEMENTS 

As  can  be  seen,  sub-elemental  inputs  to  each  of  the  primary  blocks  are  also  depicted. 
For  example,  sub-elements  for  the  determination  of  aircraft  support  are:  the  tanker  aircraft 
support  requirements,  for  added  fuel  not  normally  available  to  limited  range  fighter  aircraft; 
maintenance  support  availability  and  maintenance  status  of  the  aircraft;  reliability  analysis  to 
determine  aircraft  capabilities  to  perform  specific  types  of  missions;  and  pilot  availability  (to 
include  pre-/post-mission  crew  rest,  certification  in  type  aircraft,  and  turn-time  to/from 
other  mission  activities). 

Only  partial  development  of  the  BB  system  was  pursued  beyond  prototyping,  due  to 
lack  of  GURU  shell  documentation  for  password  protection/cluster  interface,  and  database 
development  delays  for  such  uncertainties  as  weather,  aircraft  and  instrumentation  reliability, 
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etc.  The  RESIDS  menu  options  will  support  BB  input  forms  and  it  is  anticipated  that  other 
capabilities  such  as  resource  status  analysis,  creating  and  updating  BBs,  evaluating  Wing  goals, 
projecting  daily  and  monthly  activity  rates  can  be  implemented. 

BBPREM1 


Figure  3 

EXAMPLE  FLOW  CHART 
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Rule-Set  Development  Example 

Figure  3  depicts  an  example  flow  chart  used  in  rule  set  development  for  the  RESiDS 
goal  of  including  BBs  into  the  RESOMS  scheduling  process.  [NOTES;  (1)  BBPREM  is  the  "coding" 
word  for  "blackboard  premise."  (2)  The  flow  chart  depicted  was  used  versus  the  decision  tree 
GURU  produces,  for  ease  of  presentation  in  this  paper.  It  represents  only  one  elemental  branch 
of  the  RESIDS  model  --  BBPREM  #8  -  and  includes  only  those  rules  and  certainty  factors 
applicable  to  Aircraft  Requirements  determination.  (3)  BBPREM1 -BBPREM7  are  not  discussed.) 
The  flow  chart  is  easy  to  follow  but  of  special  note  is  the  Aircraft  Reliability  Certainty  Factor 
(cf)  is  included  to  tag  with  the  remaining  determinates.  Also,  note  certain  occurances  of  a 
"NO"  will  still  lead  to  a  "TRUE"  for  the  BB  Goal.  This  situation  will  occur  if  an  aircraft  is  not 
required  since  the  mission  may  be  a  ground  test  and  not  an  airborne  test. 

Case  Study  Conclusions  and  Recommendations 

The  practice  of  submitting  BBs  to  accomplish  "special"  taskings  will  not  disappear; 
however,  it  is  reasonable  to  assume  the  use  of  RESIDS  will  alleviate  some  of  the  burden.  It  is 
anticipated  that  the  implementation  of  RESIDS  will  assist  management  by  decreasing  the  time 
required  to  research  and  analyze  BBs  being  submitted  to  them  for  consideration.  In  fact,  it  is 
anticipated  the  time  required  to  perform  the  review/pre-approval  functions  of  BBs  can  be  cut 
in  half,  saving  approximately  3-5  hours  per  day.  Further,  once  the  uncertainty  and  fuzzy 
variable  databases  are  on-line  and  accessible  to  RESIDS,  the  time  required  for  analysis  will  be 
substantially  reduced.  Finally,  by  using  RESIDS  to  expedite/expand  decision  making  capabilities 
and  to  develop  scheduling  consistency  (through  the  eyes  of  the  customer),  the  likelihood  of 
customer  satisfaction  may  facilitate  return  business. 

STIC^M-oriented  firms  who  view  information  as  a  major  economic  resource  that  is 
a  desirable  investment  rather  than  as  merely  a  necessary  expense  can  far  exceed  their 
competitors  in  the  market.  Not  only  will  they  be  leading  the  pack  in  growth,  they  will  be 
utilizing  advanced  technological  developments  that  will  assist  with  future  visioning.  Thus, 
managers,  by  studying  the  processes  that  cause  information  to  be  needed,  gathered, 
manipulated,  stored,  communicated,  and  used  (from  a  STIC^M  perspective),  can  develop 
sKaiegies  and  tools  (such  as  expert  systems)  to  make  themselves  or  their  organizations  more 
effective,  efficient  and  productive  in  producing  the  quality  products  their  customers  desire. 
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